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ABSTRACT 


Tne  Delphi  Project  is  to  develop  improved  procedures  for  the  use 
of  expert  judgement  in  decision-making  through  computer-based 
on-line  decision  structure  v/hich  can  be  employed  to  formulate 
policy  problems  and  select  preferred  policy  options.  The  laboratory 
configuration  was  completed  and  a  series  of  experiments  were 
conducted.  The  next  progress  report  will  be  published  1  Jan  74. 


'llio  I 'el, 'hi  1' 
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riject  !\as  ar  it  6  cant  i  nu  ini’  honl  the  development 
procedures  for  the  use  of  ex])ert  iudi’mont  in  decision- 
long- ranp.e  program  to  iiuplcncnt  this  aim  in  the 
is  the  development  of  a  prototype  on-line  decision 


structure  which  can  be  employed  by  governnental  agencies  or 
industrial  decision-making  groups  to  formulate  policy  problems 
and  select  preferred  policy  options. 


fho  ovicntatioii  of  the  rirojoct  is  jirimarily  experimental  and 
technological,  i.e.,  the  laboratory  testing  of  promising  innova¬ 
tions  in  group  judgment  procedures.  In  addition,  about  ZOi  of 
the  effort  is  expended  in  developi.ng  related  analytical  tools 
and  in  pursuing  theoretical  developments  Kldch  can  further  the 
experimeri  t  al  nrogram. 


Activities  through  June  1975 

Laboratory  l-xperimen 

The  laboratory  configuration  for  running  Delphi  experiments  was 
completed  during  thi<  period,  and  a  series  of  experiments  was 
conducted,  using  upper-class  and  graduate  students  at  UCLA  as 
subjects.  The  experiments  were  concerned  with  the  utilization 


)!  id'l  i  t  ;  in  1 1  (i'actu;.!)  i  ri  I  u  rma  t  ion  by  the  subjects  in  revising 
'hi  ’f  esrinitos  of  1 1  na.iac  tyne  liucstions. 

r'lc  practical  import  of  this  research  arises  from  the  fact  tliat 
in  traditional  procedures  for  applying  group  judgment  to 
govenmental  and  industrial  decision  making  (panels,  committees, 
staff  studies,  etc.)  a  major  part  of  the  activity  of  the  group 
ii  taken  up  by  assimilating  information  v.’hich  is  not  "in  the 
heads"  of  tlie  group  when  it  is  convened.  The  assimilative 
activity  involves  collecting  and  reading  additional  books  and 
reports,  site  visits,  briefings  by  otiier  e.^perts,  and  the  like. 

Most  of  these  activities  are  not  compatible  with  Delphi  procedures. 
Geographic  dispersal  of  panels  and  use  of  mailed  questionnaires 
orcclude.s  transmitting  large  amounts  of  information  to  each  member 
-if  the  group;  for  many  Delphi  studies,  the  panels  are  large  (50  or 
i-'Ore  ir’cmbers]  which  rules  out  many  of  the  traditional  information 
handling  5)rocedures  such  as  face-to-face  discussion;  and  generally, 
Delphi  studies  encompass  a  wide  range  of  subject  matter  which  makes 
tlio  relevant  additional  material  unmanageably  large.  It  has  been 
an  underlying  assumption  of  many  Delphi  practioners  that  using  a 
La.rge  group  of  c.xpert  respondents  assures  that  most  of  the  relevant 
information  is  already  in  the  heads  of  the  respondents,  or  readily 
available  to  them  as  individuals. 

There  are  reasons  for  suspecting  that  this  assumption  is  not 
justified.  It  is  clear  from  the  amount  of  effort  assigned  to 


I  ii:  I  I  c  c r  la  f  f  h  I,  •!  r  ;  V  :  i 

'  ‘  V. .  1,1,11  I  ri  I  s  aciLVirv  i  considorcd 


1  in-  one  of  tht-  nosr  impuil^n'  c Uinonont s  of  tbo  oronp 

‘-‘^•^^*^’'^11,  the  rather  .sp.-irsc  experimental 
thu  e.Msts  Oil  the  .subject  inuicutes  that  utilization  of 
-Tcldjtional  1  actual  in  fo  rma  r  i  on  rn„  increase  the  accuracy  of 
y.roun  judgment  by  a  large  factor. 


T!,e  development  of  procedures  for  doalinp  Kith  additional 
factual  infornatton  which  can  approximate  the  desirable  feature 
of  t!:e  traditional  "solf-oducation”  activities  of  committoes, 
and  at  the  same  time  can  be  utiHted  in  a  Delphi  context  is 
thus  one  of  the  crucial  areas  where  additional  capabilities 
are  needed.  The  experimental  series  besan  this  spring  is  aimed 
at  obtaining  data  which  win  be  useful  for  deslgnins  and  testini 


liuch.  techniques. 


In  carl  Lor  c.\pe r iincn t.s  conducteo  at  RAND,  feed-in  of  additional 
factual  ruiterLal  was  identified  as  one  of  the  no.st  effective 
procedures  lor  iriprovins  both  individual  and  group  responses/ 
ivhen  facts  were  fed  in  in  random  order,  on  the  average  there 
vas  a  decreasing  return  for  each  additional  fact.  In  the  preser 


Studie^^^ 


( 


■-■'lit.,  vf  -irc  presontcHi  Kith  •,  mc-nu  of 

■  our  pjt-.uir:  11  ficts  (thut  is,  tho  fjicts  ,,ix-  df-GoriSud  but 
tuner,,-:,!  value-,  arc  not  stated.  I.  The  s-.bjoct.s  rate  each  of 
tile  four  i„  terns  of  thei,  ex.occted  value  in  improvinR  the 
■•n.sKer  to  the  prinar)-  question.  The  facts  arc  then  presented 
11,  one  of  t,(o  orders;  individually  to  each  subject  in  the 
order  of  his  ratine,  or  collectively  to  .all  subjects  in  the 
order  of  the  group  average  rating. 

To  dare,  four  experimental  sessions  involving  10  subjects  per 
session  and  10  primary  questions,  have  been  run  in  the  individua 
.selection  mode,  and  five  sessions  varying  from  five  to  10 
subjects,  and  four  to  10  questions  have  been  run  in  the  group 
selection  mode.  Preliminary  analysis  of  the  data  indicates 
that  the  hypothesis  that  error  uill  decrease  much  more  rapidly 
under  individual  .selection  of  facts  than  under  random  presenta¬ 
tion  is  homo  out.  In  addition,  the  decrease  in  return  per 
additional  fact  is  greater  with  self-selection. 


First  returns  on  the  group  selection  m.ode  indicate  that  the 
decrease  in  error  is  even  more  rapid  under  grour.  selection; 
however,  the  data  is  less  extensive  than  for  the  individual 
mode,  ind  appears  to  be  interacting  with  the  feedback  of 
medians  and  modes  of  the  first  round  answers.  (In  the  RAND 
experiments,  stati'tical  feedback  in  conjunction  with  additional 
iacts  did  not  appear  to  have  a  significant  effect  on  group  erroi 
In  the  present  experiment,  it  appears  that  it  might.  This  could 


I  liil  i  1. .!  r  >: 


Llif  ;•.<  i.  s  t  v'.u'c  of  f  i'odback  ,  feocl-in  iritrrnctinn  in 
t'.'c  (.‘ase  :  c  t  ;U-  !  act  j  aro  ao ’..oatocl ,  than  ocinj' 

prt‘>c..t:'ad  at  rniuioin.) 

The  accompanyinr.  figure  shows  the  prciininury  icsults  concern¬ 
ing  teed- in  of  facts.  The  upper  curve  shews  the  results  for 
random  feed- in  for  comparison. 

fn  experimental  sessions,  the  CCDS  interactive  system  has 
worked  smoothlv  and  response  time  of  the  system  has  been  well 
within  the  tolerance  level  of  subjects.  The  interactive  features 
of  the  system  have  been  crucial  in  conducting  tiir'  P.voup  selection 
mode  of  tiuj  feed- in  of  additional  material,  and  hence,  the 
experiments  have  shown  the  value  of  the  system  for  conducting 
more  coiuplex  types  of  Pelphi  exercises. 

Cine  un..  ii !.  i  ci  pated  effect  of  the  present  configuration  is  the 
relatively  slow  pace  of  the  group  in  completing  a  cycle  of 
responses  to  a  given  question.  This  appears  to  be  almost 
entirely  a  result  of  the  large  variation  in  individual  response 
times  for  tlic  various  subtasks.  It  is  possible  that  the  complete 
isolation  of  individual  subjects  in  separate  rooms  contributes 
in  ti’is  variability,  and  si.hscqiicnt  experiments  w’ill  try  more 
"open"  configurations  to  allow  a  measure  of  group  self-pacing. 

It  is  hoped  tliat  this  tactic  will  increase  tlie  amount  of  data 
collected  v.'ithin  a  given  oxpcriricntnl  session. 
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AL..MTlLy\Al.  1-ACTS 


Kandoa  Order 


Individually 
N  selected 


Group 

Selacteci 


NUT3ER  OF  FACTS 


“‘Error  on  round  one  (no  facts)  =  1. 


la 


0 


i''  to  ret) !  i  ca t  i iig  some  of  tJic  expe  r  i.i:ient.s  concerned 

-izr  feed- ip.  .>j  reJr-'.ppc  facts  (p  r  i  inir  LJ  y  to  increase  the 

o'  j-it  I  to  generate  an  acceptable  level  of  significance) 
sii'iiar  experimetits  will  be  tun  during  the  fall  where  the  mode 
ol  response  will  be  a  probability  distribution,  rather  than  a 
point  estimate.  These  experiments  will  furni.sh  data  to  assess 
the  leliability  of  probability  estimates,  to  evaluate  the 
Cl  feet  of  factual  feed-in  for  probability  esti.^ates,  and  to 
test  several  hypotheses  concerning  the  dependence  of  anonymous 
probability  judgments. 


Although  at  the  moment  the  data  is  not  sufficient  to  allow 
finalizing  technicpies  for  integrating  external  information 
in  an  applied  Dolplii  exercise,  a  "first  approximation"  appears 
to  he  emerging  from  tlic  experimental  results.  A  summary  of  this 
hypothetical  .set  of  procedures  is  perhaps  the  clearest  way 
to  show  the  [)ractical  import  of  the  relevant  fact  experiments: 

(1)  The  arst  judgmental  round  of  a  Delphi  exercise  should  be 
run  in  the  "standard"  way  --  i.c.,  anonymous  judgments  from  the 
panel,  using  only  the  information  in  their  heads,  or  what  is 
conveniently  incorporated  into  the  ffues tionnairc  itself,  (2) 

The  first  round  questionnaire  should  include  an  open-ended  sot 
of  re.sponsos,  where  each  export  is  asked  to  list  information 
vdiich  b.c  thinks  Is  available,  feasible  to  obtain  in  a  reasonable 
time,  and  is  important  for  increasing  the  certainty  of  the  group 
iudgment.  (5)  For  those  questions  where  there  is  significant 


disanreoncnt  on  the  first  round  (significant  in  the  statistical 
sense)  and  where  the  list  of  desired  information  from  the 
respondents  is  highly  diverse,  an  auxiliary  round  is  run  to 
obtain  Judgments  of  relative  importance  of  the  nominated  informa¬ 
tion.  (4)  Questions  with  high  agreement  on  the  first  round 
are  not  iterated.  Questions  wi’th  disagreement  on  the 
answer,  but  reasonable  agreement  on  the  needed  information,  are 
Iterated,  with  as  much  of  the  nominated  information  as  can  be 
obtained  included.  for  questions  with  disagreement  on  both 
answers  and  desired  ..formation,  the  results  of  the  auxiliary 
judgments  concerning  importance  and  feasibility  are  used  to 
select  ti;e  items  of  information  which  are  researched  and  fed 
into  the  groups.  (5)  For  those  cases  where  the  nominated 
information  is  relatively  extensive,  a  division  of  labor  is 
designed,  based  on  the  individual  judgments  of  importance,  where 
c-he  relevant  infc  rmation  is  apportioned  among  the  members  of 
the  group  in  roughly  equal  proportions. 

During  the  research  (collection)  phase  of  this  procedure,  a 
quick  round  may  be  intercalated  to  determine  if  one  or  more 
members  of  the  group  know  the  specific  information  requested, 

or  can  obtain  it  easily.  Here  ‘a  standards  for  "know"  would 
be  very  liigh. 

Ibis  structure  has  not  yet  been  tested.  It  will  be  tested  in 
several  exercises,  beginning  in  the  fall  of  1975. 
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! ' ^  1  ^  ^ i on  of  TRAC li  f o  f  Data  Vm  lysis 

An  exercise  i:.  under  way  to  assess  the  potential  role  of  the 
1 RACL  systeiu  for  analysis  of  the  data  base  being  generated  by 
the  Delphi  experiments.  The  TRACI:  data  manipulation  capabilities 
are  well  suited  for  the  complex  and  multiply  nested  data  structures 
produced  by  Delphi  exercises.  Development  of  a  convenient  way 
to  routinely  process  the  data  by  TRACE  should  increase  the  speed 
and  depth  oi  analysis  by  a  large  factor.  This  becomes  especially 
important  in  studying  the  "fine-grain"  effects  of  order  of  facts, 
fact  ratings,  and  the  like. 


1 
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Design  of  Formatted  Conference 


The  initial  design  has  been  completed  for  a  software  system  that 
wiJl  allow  a  geographically  dispersed  conference  (e.g.,  via 
the  AREA  iVetwork)  to  be  conducted  on  a  variety  of  decision 
problems.  The  software  system  is  described  as  a  formatted 
conference,  to  distinguish  it  from  a  number  of  oth.er  computer 
conferencing  systems  tliat  operate  in  a  more  open-ended  fashion 
(e.g.,  the  conferencing  system  developed  at  OEP  by  Murray  Turoff.) 

The  system  embodies  a  Iiighly  general  model  of  the  problem 
identification  and  group  decision  process:  specification  of 
objectives  or  goals,  specification  of  policy  alternati’''^es ,  and 
judgments  of  the  expected  contribution  of  alternatives  to 
objectives.  The  major  ii’.etiiodolog ical  problem  facing  such  a 
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group  process  Is  the  l  orniu lat ior.  of  techniciuos  for 
reducing  the  highly  diverse  attitudes  and  points  of  view  of 
diliereiit  mciribers  of  a  panel  to  a  coherent  and  manageable 
structure,  [n  the  prototype  decision  system  under  development, 
major  reliance  v/ill  be  put  on  dimens ion- reduction  techniques 
such  as  multi -dimensional  analysis,  cluster  analysis  and  factor 
analysis.  As  of  the  moment,  it  looks  as  if  cluster  analysis 
will  be  the  most  useful  of  these  techniques,  but  the  system  vdll 
incorporate  the  capability  to  use  all  three,  thus  allowing 
comparison  of  their  relative  value  in  the  shakedovm  exercises. 

Initially,  it  is  being  assumed  that  identification  of  policy 
alternatives  can  be  carried  out  in  much  the  same  way  as  identifica¬ 
tion  of  goals  and  objectives;  thus  a  common  group  process  is 
being  set  up  for  tlicse  two  activities.  The  process  consists 
of  the  iollowing  basic  steps:  (1)  Listing  by  eacli  panel  inesiber 
of  the  inoSi  important  goals  and  objectives  (or  policy  alternatives) 
for  the  present  decision  problem  as  he  perceives  them.  (2)  Formula¬ 
tion  of  a  master  xist  of  the.se  individual  items.  (3)  Group 
editing  ol  the  items  in  the  master  list,  including  ratings  for 
clciiit)',  for  relevance  (or  importance)  aiid  for  identity  among 
items.  (d)  V/ithdrawal  or  rewriting  of  unclear  items,  and 
consolidation  of  identical  items.  (5)  Sorting  by  each  individual 
of  the  euited  list  into  what  he  perceives  as  similar  categories. 

(6)  Use  of  a  cluster  routine  (or  multi-dimensional  scaling  or 
factor  analytic  routine)  to  aggregate  the  individual  classifica¬ 
tion  and  formulate  a  (much  shorter)  list  of  aggregated  goals  and 


oh  j  cct.  i  \'L' :  (ov  policy  al  t  criin  t  i  ves)  . 

The  sofn/ire  systom  should  be  runniiu-.  in  prototype  Ln  the  Pall 
of  1P7.5.  Shakedown  exercises  will  investigate  both  the  efficiency 
ol  the  system  as  a  method  of  decision  formulation  and  thq  meaning¬ 
fulness  of  the  aggregated  objectives  and  policy  alternatives 
to  the  group  members.  By  the  spring  of  11)74,  the  system  should 
by  ready  for  a  pj.lot  study  with  either  a  governmental  agency  group, 
or  an  industrial  contractor.  The  selection  of  an  appropriate 
client  will  be  made  in  cooperation  with  ARPA. 

Other  uses  of  the  decision  analysis  system  are  envisaged  in 
addition  to  direct  use  in  decision  exercises  by  operating 
agencies.  The  output  of  system  can  be  used  in  a  number  of 
experimental  studies  of  basic  aspects  of  group  decision-making. 

One  of  the  important  features  of  cluster  analysis  routines  is 
that  tiiey  generate  a  hierarchy  of  items  at  different  levels  of 
generality.  ihit  shortly,  the  system  will  generate  a  set  of 
nested  decision  models  of  various  levels  of  abstractness.  Such 
a  sot  of  models  will  enable  the  testing  of  hypotheses  concerning 
the  optimal  level  of  detail  for  formulating  decision  problems, 
as  a  Function  of  the  degree  of  uncertainty  of  the  group  concerning 
basic  elements  of  the  problem.  Similarly,  they  will  enable 
testing  of  various  hypotheses  concerning  the  optimal  v.'ay  to 
combine  indices  of  uncertainty  (dispersions  or  self  ratings) 
for  complex  judgments. 


11 


Anal y  ^  i  r  l')cve  I  o;)menti; 


Ci-oss- Impact  matrices.  In  the  previous  semi-annual  report,  a 
basic  development  in  the  theory  of  cross-impact  matrices  was 
described.  This  is  formulation  of  a  distance  measure  whereby 
a  set  of  probabilities  can  be  mapped  onto  a  metric  space. 

Since  then  a  beginning  has  been  made  in  treating  the  theory 
of  cross-impacts  as  a  function  of  time.  The  elementary  theory 
has  been  fully  developed,  but  is  too  complex  for  event  sets  of 
the  size  met  with  in  practice.  i/ork  is  continuing  on  the 
derivation  of  a  reasonable  approximation  for  large  sets  of  events. 


Probability  Aggregation.  The  analysis  of  group  judgment 
define  I  as  an  aggregation  of  probability  judgments  by  individuals 
has  been  extended  by  application  of  probabilistic  scoring  rules 
to  several  potential  aggregation  methods.  The  major  outcome  of 
this  analysis  is  that  if  the  aggregation  rule  is  a  statistical 
composite  (a  mean,  geometric  mean,  or  the  like)  then  the 
probabilistic  score  of  the  cemposite  is  always  greater  than  or 
equal  to  the  average  probabilistic  score  of  the  individual 
members;  if  the  aggregation  rule  is  an  optimal  estimate, 
employing  Bayes'  rule  or  an  inverse  Bayes'  rule,  then  the  net 
probabilistic  score  (defined  as  the  score,  minus  the  score 
that  would  be  obtained  by  using  the  a  priori  probabilities  as 
an  estimate)  is  given  by  the  expression  5(0)  =  nSXT)  +  D,  where 
S(rO  is  the  net  score  of  the  group,  ^XTT  is  the  average  net 
score  OL  the  individual  members,  n  is  the  number  of  lembers  of 


me  gr^up,  and  H  is  tJie  expected  dependence  among  the  individual 
estimates.  ]i  D  is  small  or  positive,  and  the  average  member 
of  the  group  has  a  positive  net  score,  then  the  group  does  "n 
times  as  well"  as  the  average  individual:  if  on  the  other  hand, 
the  average  net  score  of  the  individual  members  is  negative, 
then  the  group  does  "n  times  as  bad"  as  the  average  individual. 


These  results  have  been  incorporated  in  a  revised  version  of 
iM-a/  "Delphi ,  Some  Basic  Considerations".  Some  of  the  up¬ 
coming  experiments  with  probabilistic  judgments  will  be  used 
to  goncrato  data  that  will  identify  condicions  under  which  the 

optimstic  type  of  result  can  be  expected  to  occur  with  probabilistic 
aggregation. 


with  Feedback.  A  mathematical  model 
has  been  developed  to  describe  the  change  in  group  distributions 
of  point  estimates  bctwe<m  round  one  and  round  tv.^o ,  when  the 
group  receives  feedback  of  medians  and  quart: les  from  round  one. 
Analysis  of  data  on  individual  estimate  change  indicates  that 
a  good  approximation  is  the  individual  shifts  toward  the  median 
a  constant  fraction  of  his  distance  from  the  median  on  the 
i-irst  round;  i.e.,  for  a  distribution  where  the  median  =  1, 


•  I  - 


x(]-s)  +  s 


where  x'  is  the  second  round  estimate,  x  is  the  first  round 
esii.iiatc,  and  0  ;<  s  ;<  1  is  the  fractional  change.  From  extensive 
data  on  individual  changes,  s-^.6;  i.c.,  the  individual  does  a 
little  more  than  "split  the  difference"  between  his  location 


-  - 


iiiiJ  tp.it  of  Lho  iMO'Iiiiii. 


1.1P  son,i:d  part  of  tlic  modol  is  the  nssuisption  that  the  first 
vouad  distrihution  is  los  aormal.  This  assumption  has  been 
'veil  verified  by  previous  studies.  The  seometric  mean  on  the 
second  round  (GM')  uould  be  predicted  to  be  Cfor  a  log  normal 
distribution  Kith  mean  of  the  log  transform  =  o  and  standard 


lieviation  =  1.) 


i~(  ) 


for  s  =  .6,  the  equation  predicts  an  increase  of  Ill  i„  the 

.geometric  mean  botKeer.  round  one  and  round  tuo.  This  is  slightly 

snaller  than  the  observed  increase  of  about  151.  At  the  moment 

it  i.s  not  Clear  whether  this  discrepancy  is  due  to  departures 

from  the  assumption  of  Ugnormality  on  the  part  of  specific 

distributions  for  specific  questions,  or  whether  there  is  some 

additional  factor  leading  to  improvement  on  feedback.  The 

model  ignores  some  of  the  change  behavior  in  the  vicinity  of  the 
median . 


Additional  Activi ties 


Morman  Dal  key  participated  in  a  UNESCO  working  conference  on 
Man  and  the  Biosphere  at  UNESCO  Headauarter^ 


Erance,  during  the  vu.-f'k  of  March 


leatiquarters ,  Paris, 

26-30,  19/3,  where  he  presented 
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a  paper  on  "State  of  the  Art  in  Mea'^ureinent  of  Quality  of  Life" 
anu  ciiuired  the  v-'orking  group  on  methodology  in  research  on 
cnvironm  ntal  perception.  He  has  acted  as  an  informal  consultant 
to  the  Public  Health  Service  fNTM)  on  evaluation  of  research 
proposals,  and  to  the  Air  Ftesources  Board  of  the  State  of 
California  on  a  Delphi  evaluation  of  disability  resulting  from 
types  and  amounts  of  air  pollutants.  He  also  consulted  with 
the  Air  Force  Office  of  Medical  Personnel  on  a  service-wide 
Delphi  study  of  medical  personnel  problems  arising  from  the  all¬ 
volunteer  policy. 

Forthcoming  Reports 

1.  Elementary  Cross  Impact  Model  Including  Dependence  on  Time. 

2.  A  Model  of  Croup  Judgment  Change  as  a  Function  of  Statistical 
Feedback . 

Reports  Issued 


TM-37  Delphi:  Some  Basic  Considerations. 

TM-42  A  Delphi  Study  of  Factors  Affecting  the  Quality  of  Life. 


